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ABSTRACT 

Artificial intelligence (Al) techniques and virtual 
reality (VR) make possible powerful interactive stories, and this 
paper focuses on examples of virtual characters in three dimensional 
(3-D) worlds. Waldern, a virtual reality game designer, has theorized 
about and implemented software design of virtual teammates and 
opponents that incorporate AI techniques including fuzzy logic, 
neutral nets, and genetic algorithms. He asserts that one of the 
primary goals of VR is to generate virtual actors with whom the 
participant can interact in a contextually meaningful fashion, and 
proposes a classification scheme for such characters, as determined 
by intelligence level and centrality to the game. Virtual characters 
that can reason about their environment in original ways are being 
developed. PLACEHOLDER, a VR art project inspired by Native American 
myths and stories, features' virtual characters where VR explorers can 
play the role of a bird, snake, person, or other creature. 
Participants, embodied in "smart costumes," take on the animal's 
pattern of movement, character, physical abilities, and may explore 
different sites or mark home territory. Other examples of virtual 
characters are: Dolby's "Virtual String Quartet," which synchronizes 
animated figures with 3-D sound; and an "avatar," a virtual character 
that reflects a person's look, mood and personality, and who 
represents him or her in a virtual world. (Contains 8 references.) 
(MAS) 
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We've always had virtual worlds: 
virtual worlds for the imagination to enter. 
Stories, myths, movies. From the campfire 
to the silver screen, and npw the computer 
screen. Stories are one of the most 
fundamental and powerful synthetic 
experiences available to us. Interactive 
stories are changing our relationship to 
information. And most recently, the addition 
of artificial intelligence techniques to virtual 
reality extends our ability to create 
interactive, enacted stories with characters 
that perform in powerful new ways. 

Artificial intelligence researchers 
such as Roger Schank and John Seely 
Brown have increasingly focused on 
stories. For example, Roger Shank has 
extended his study of scripts for everyday 
events, such as placing an order in a 
restaurant, to cover full-blown stories. 
Scank has concluded that people have 
expectations concerning story structure, 
what he calls 'archetypes. 1 

Can you do the walk? Can you do 
the talk? Can you improvise? Virtual 
characters are increasingly able to do this 
with the addition of artificial intelligence 
techniques and powerful modeling of 
human (and animal) behavior. With artificial 
intelligence features, it's not just the 
content, but interaction with other 
characters, both real and computer- 
generated, that are adding to engagement 
and immersion. 

We can think of virtual characters as 



a form of puppet. Puppetry is the theater of 
manipulated objects. Virtual characters are 
calling their own shots, based on the 
programming that's embedded in them. 
There are wonderful 2-D virtual puppets, 
including a Virtual Mark Twain and a Virtual 
Mario, that are controlled directly by human 
puppeteers — sort of like stand up comics — 
- but this paper will focus on the characters 
in 3-D worlds. 

One designer has developed a Virtual 
Globe Theater where virtual actors act out a 
scene from Romeo and Juliet. You can have 
any seat in the theater, even a bird's eye 
new. The setting is highly interactive (and 
historically accurate). However, the virtual 
actors are just going through a programmed 
routine; there's no interactivity or 
improvisation. This is a wonderful 
application, but there are even more 
'intelligent' virtual characters. 

Virtual Characters in Games 

Virtual reality game designer 
Jonathan Waldern (1994) has theorized 
about — and implemented — - software 
design of virtual teammates and virtual 
opponents that incorporate AI techniques, 
including fuzzy logic, neural nets, and 
genetic algorithms. According to Waldern, 
one of die primary goals of VR is to 
generate software constructs or 'entities 1 
(virtual actors) with whom the participant 
can interact in a contextually meaningful 
fashion. Waldern reports that the essential 
characu^iistics of a virtual actor are: (1) They 
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have goals and motivations, not all of which 
are necessarily apparent to the participant; 
(2) They may be observed to receive 
sensory input both from the virtual world 
and from the participant; and (3) They have 
the capability to learn and hence adapt to any 
aspect of their environment. 

Waldern proposes a classification 
scheme for different types of virtual actors, 
based on what AI characteristics are 
emphasized: (1) V-Extras; (2) V -Agents; (3) 
V-Actors; (4) V-Androids; and (5) V- 
Experts. Each category is progressively 
more "intelligent" and more central to the 
action in the game. For example, a V -Expert 
is an entity that may evolve it responses to 
events based on observation and experience 
within a virtual environment. 

The sof tware implementation of 
Waldern's virtual actors features several AI 
techniques: fuzzy logic, neural nets, and 
genetic algorithms (Waldern, 1994). In 
ordinary logic, a proposition is either true or 
false, but with fuzzy logic, a proposition is 
given a probability value between zero and 
one. A neural net consists of a two- 
dimensional array of neuron elements, each 
of which receives an input from its 
neighbors when they are activated, creating 
a network for sending signals. This is 
similar to the characteristics of living 
organisms in terms of ability to correlate 
ambiguous data and ability to learn by 
comparing outputs to a set of known 
results. Genetic algorithms are a powerful 
evolutionary technique whereby the 
population of problem-solving algorithms 
(strings of code) are permitted to breed and 
mutate so that when a problem is 
encountered, the population is evaluated to 
assess the quality of the solution each 
algorithm offers. By employing AI 
techniques, Waldern (1994) explains that it 
should be possible to create entities that are 
entertaining and instructive to communicate 
with. 

Virtual Characters in Research and 
Development 

A group of researchers led by 
Norman Badler, at the Center for Human 



Modelling and Simulation at the University 
of Pennsylvania, are creating simulated 
characters that can reason about their 
environment in original ways. Taubes 
( 1994) reports that the center's main project 
is a computer model designed to mimic 
human movement. This model is used by- 
engineers to test com puter-desi gned 
products including bulldozers, buildings, 
and helicopter cockpits. Scientists in the 
center's "synthetic -conversation group" 
teacli the computer-simulated humans the 
rules of speech, facial expression, and 
gesture. Virtual characters like Gilbert and 
George can then interact with minimal 
human oversight, improvising based on 
information they have been supplied with. 
In one exchange between a bank teller and a 
customer who wants to withdraw fifty 
dollars when there is only three dollars in 
his account, the banker is able to determine 
that there's not enough money to make the 
withdrawal. Badler suggests that these 
characters will come to look more and more 
like us, behave like us, and respond to 
language and stimuli as we do within the 
confines of a 3-D virtual world (Taubes, 
1994). 

Virtual Characters in Art 

The example of Gilbert and George 
models logic and human movement what 
about mood and personality? Bates (1992) 
argues that it is not sufficient to solve for the 
physical dynamics and movements; one 
must also solve for the culture. Different 
cultures have very different ways of 
telegraphing intention and mood through 
body language, so ideally these will be 
captured. 

According to Bates (1992), "Most 
existing research on virtual reality concerns 
issues close to the interface, primarily how 
to present an underlying simulated world in 
a convincing fashion. However, for virtual 
reality to achieve its promise as a rich and 
popular artisitc form, as have the novel, 
cinema, and television, we believe it will be 
necessary to explore well beyond the 
interface, to those issues of content and style 
that have made traditional media so 
powerful, (p. 133)" 
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Bates reports that almost always, 
television, movies, and novels include at 
least three other key elements besides the 
physical space: "First, there are living 
creatures, usually human, and usually 
embodying some intelligence and emotion. 
These let the viewer see the world as a place 
of life, purpose, and feeling. Second, there 
is long-term structure to the events 
portrayed, which is to say that some kind of 
story is told. The story gives intensity and 
meaning to the world. Finally, the world is 
presented in an effective, emotionally 
powerful style. Cinematic and narrative 
technique are highly developed examples of 
the art of presenting worlds, (p. 134)" 

As Bates points out, traditionally, 
very little research and development in 
virtual reality focused upon computational 
models of cognitive-emotional agents, long- 
term dramatic structure, and presentation 
style — essential features for a successful 
artistic medium. But this is changing. 

For example, in a VR art project that 
research artist Brenda Laurel helped to 
design, inspired by Native American myths 
and ceremonies, VR explorers can play the 
role of a bird or a snake or another creature, 
taking on the pattern of movement and the 
vantage point of that animal (Frankel, 1994; 
Laurel, 1994). This is the PLACEHOLDER 
project that she developed with Rachel 
Strickland. PLACEHOLDER was funded 
by Interval Research and the Banff Center 
for the Arts. 

According to Laurel (1994), science 
is a dialogue with nature. She comments, 
"Some patterns and rhythms can only be 
appreciated through analysis. Both science 
and art are about the natural world and the 
unseeable. Art deals with human nature. 
Making art with computers is so very hard 
to do because it is so cut off from nature. 
Virtual reality is the antithesis of this." 

PLACEHOLDER is an artistic 
research project inspired by topography as 
well as Native American myths and series. 
PLACEHOLDER features a virtual world 
that is inspired by the landscape of several 



locations in the vicinity of Banff National 
Park in Alberta, Canada. These sites include 
a sulphur hot spring in a natural cave, a 
waterfall in a canyon, and a formation of 
hoodoos (tall, narrow rock formations or 
pillars) overlooking a river. Laurel 
comments, "The cave was the place we 
began for obvious Jungian reasons/' In 
addition to the virtual world, there is a map 
of the PLACEHOLDER site that looks like a 
Native American graphic or rock painting. 

This application can accomodate two 
people wearing head-mounted displays 
(who may be physically remote). It is 
designed to support exploration and play. 
People can walk around, speak, and touch 
and move virtual objects within the 
PLACEHOLDER world. 

According to Laurel, people (as well 
as animals) mark the spaces that constitute 
their home turf. The following questions 
were asked in conceptualizing this project: 
How do people mark places? How can 
people mark places for imaginative play? 
People sometimes leave marks in natural 
places — pictograms, petroglyphs, grafitti, 
or trail signs for example. One group, the 
Anastazi people of the southwest marked 
places with rock carvings. The 
PLACEHOLDER project features markers - 
- Placeholder?. These Placeholders are in the 
form of voices. In PLACEHOLDER, people 
can leave Voicemarks - bits of spoken 
narrative - that can be created, listened to, 
and rearranged by anyone who passes 
through. These messages and comments 
within the that virtual world then serve as 
story starters. The virtual landscape 
accumulates definition through messages 
and storylines that participants leave along 
the way. The people who visit 
PLACEHOLDER will change it Laurel 
reports that one goal of the 
PLACEHOLDER project is to foster new 
forms of narrative play. 

Laurel (1994) reports that in the 
PLACEHOLDER project, the designers 
tried out different camera positions in an 
attempt to provide a dialogue that featured 
Cubist qualities and Constructivism. These 
were not a good match for the subject 
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Realism and impressionism proved to be 
better matches so these artistic traditions 
informed and inspired the PLACEHOLDER 
project. 

According to Laurel, immersive 
virtual reality systems often suffer from a 
design strategy featuring a "severed hand" 
and "severed head" syndrome — partial 
avatars or representations of the user's body 
within the virtual world. This kind of 
feature affects perception; it's not really 
natural. So the PLACEHOLDER project 
attempted to address this issue. Related to 
this, iconic gestures of the glove — such as 
pointing — are not that good at indicating 
the direction of intended movement; 
movement of the pelvis is much better for 
this. Also, an interface that only includes 
one hand is unnatural. So any kind of VR 
input device should include both hands, not 
just one, to optimize the feeling of natural 
interaction. For the PLACEHOLDER 
project, inexpensive "grippies" were used 
as interface devices. These are held in both 
hands and users grip them to signal 
interactions 

The issue of embodiment — seeing 
a representation of oneself or the character 
one is "playing" — was a central design 
issue in the PLACEHOLDER project 
(Laurel, 1994). There is a set of animal 
characters — crow, snake, spider, and fish. 
As Laurel explained, "Four animated spirit 
critters." These animals talk to you as you 
get closer, each with a distinctive voice. A 
person visiting the world may approach and 
assume the "character" of one of the spirit 
animals and thereby experience aspects of 
its unique visual perception, its way of 
moving about, and its voice. Laurel refers to 
this kind of character within the virtual 
world as "smart costumes." When you take 
on the smart costumes, this changes what 
your body can do. Your voice changes. 
Perception may change. For example, 
snakes can see i nfrared and 
PLACEHOLDER tries to model this — the 
space becomes brighter. Thus the critters 
function as "smart costumes" that change 
more than the appearance of the person 
within. You become embodied in "smart 
costumes." In PLACEHOLDER, the smart 



costumes are mostly about audio — since 
the spaces in PLACEHOLDER are dark. 
Explaining this darkness, Laurel (1994) 
emphasizes that the resolution of the head- 
mounted displays in VR are currently at the 
level of what is legally blind so that other 
factors like audio are very important 

To make up for limitations of visual 
resolution, designers must be concerned 
with stylistic issues. In PLACEHOLDER, 
the images of the animals are flat, iconic — 
like the simplified, symbolic image of an 
animal on a rock painting or some 
ceremonial image. The representations of the 
animals are like flexible sandwich boards — 
you can show gait, but not limbs. Limbs 
were not well defined in PLACEHOLDER 
(Laurel, 1994). 

Related to (his, the designers of the 
PLACEHOLDER project studied caref ully 
how to design the movements of the "smart 
costumes" in response to the gestures of the 
person embodying the various characters. 
For example, how should the movement of 
a crow be represented? What's a flap? The 
designers found that when users embodying 
the crow flapped their aims to emulate (and 
initiate) the crow's movement, one flap 
moved the participant very f ar — too far — 
that is, disproportionately far compared to 
what we expect from the actual flap of a 
bird's wings (Laurel, 1994). 

Researchers at the MIT Media Lab 
are studying ethology, the science of animal 
behavior, as a basis for designing animal 
characters in virtual worlds (Sheridan & 
Zeltner, 1^93). Thus, "smart costumes" 
may have potential in both the science and 
humanities curricula. Actually, the MIT 
researchers have produced an animated short 
film, Grinning Evil Death, to demonstrate 
their design of virtual characters based on 
ethology. 

The 1993 Guggenheim Soho 
Museum exhibit, "Virtual Reality: An 
Emerging Medium" featured artwork by 
several artists, including Thomas Dolby 
(Teixicra, 1994). Dolby prepared a VR 
artwork that featured a powerful interactive 
audio component as well as a visual 
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component. Dolby's "Virtual String References 
Quartet" featured the first artwork to 

synchronize animated figures with three- Bates, J. (1992, Winter). Virtual reality, 
dimensional sound so that various sounds art, and entertainment. Presence, 1(1), 133- 
appeared to originate from different points 138. 
in space. In the "Virtual String Quartet," 

viewers (participants) found themselves in a Frankel, K. A. (1994, January). A 
rehearsal space where a string quartet played conversation with Brenda Laurel. 
Mozart's Quartet No. 21. Moving to a Interactions, 1(1), 44-53. 
different location within the rehearsal space 

shifted the sound so that the sounds from Laurel, B. (1994, May 13). Art Issues in 
each instrument always appeared to eminate VR. Virtual Reality '94 Conference. San 
from that instrument. There was a further Jose, CA. 
element of interactivity: You found out by 

tickling the violin player, he really wanted to Perlin, K. (1994). Avatars. Pixel Vision, 
be a bluegrass player and his playing would (12), 46-47. 
shift in this direction when he was 

"touched." Similarly, the bass player Sheridan, T. B., and Zeltzer, D. (1993, 
wanted to be Charles Mingus — playing a October). Virtual reality check. Technology 
jazz bass line. The music underlying this Review, 96(7), 20-28. 
VR artwork was performed by The Turtle 

Island String Quartet (Teixiera, 1994). Taubes, G. (1994, June). Virtual Jack. 

Discover, 15(6), 66-74. 

Avatar Representatives 

Teixiera, K. (1994, May 13). Intel's IDEA 
Another type of virtual character is Project and the VR Art Exhibit at the 
an avatar who represents you in a virtual Guggenheim. Virtual Reality '94 
world. This extends the notion of virtual Conference. San Jose, CA. 
agents online that perform tasks sucii as 

sorting through databases to find Waidern, J. (1994, February). Software 

information on specified topics. Perlin design of virtual teammates and virtual 

( 1994) envisions this scenario: "You're at a opponents. In Sandra K. Helsel (Ed.). 

simulated conference. You and everyone London Virtual Reality Expo 94: 

else at the conference are represented by Proceedings of the fourth annual conference 

human-like "avatars" within an immersive on virtual reality. 120-125. 

virtual environment. Later that day, your 

own personal avatar roams through a virtual 

shopping mall and goes into the virtual 

stores where you like to shop in cyberspace. 

The avatar reflects your look, mood, and 

personality (or perhaps what you would 

prefer people to think are your look, mood 

and personality) (p. 46-47)." VPL Research 

pioneered this kind of system with its RB2 

(Reality Built for Two) where participants in 

a virtual meeting could playfully take on the 

avatar of a lobster or some other creature. 

Perlin (1994) recommends that we 
should design these avatars so that "they 
have personality and a certain joie de vivre." 
This is a very appealing idea, although it 
doesn ! t convey the full potential of virtual 
characters. _ 
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